Dynamic Programming-based Optimization for Audio-visual Skims

To enable a viewer to understand the original plot by just watching a condensed video, a successful video summarization system should yield a summary that preserves both content highlights and information coverage. Content highlights include a set of diverse concept patterns, each of which corresponds to semantically similar video shots and video scenes, each of which represents a sequence of highly related and temporally adjacent shots. Decent information coverage helps to eliminate the distant jump between two nearby shots in the summary. In this paper, we propose an efficient Dynamic Programming algorithm to select salient video shots as well as balance the two above criteria in a global optimization framework. In this framework, a saliency tuning method is proposed to fuse information from multimodal (visualaural-motion) features through hierarchical (scene, concept pattern, shot and sub-shot) structures. From experimental results, the performance of our approach behaves well in informativeness and enjoyability of a video summary.